Enhanced object reconstruction

ABSTRACT

Processing images includes projecting an infra-red pattern onto a three-dimensional object and producing a first image, a second image, and a third image of the three-dimensional object while the pattern is projected on the three-dimensional object. The first image and the second image include the three-dimensional object and the pattern. The first image and the second image are produced by capturing at a first camera and a second camera, respectively, light filtered through an infra-red filter. The third image includes the three-dimensional object but not the pattern. Processing the images also includes establishing a first-pair correspondence between a portion of pixels in the first image and a portion of pixels in the second image. Processing the images further includes constructing, based on the first-pair correspondence and the third image, a two-dimensional image that depicts a three-dimensional construction of the three-dimensional object.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation application of U.S. patentapplication Ser. No. 11/327,651, filed Jan. 9, 2006, which claims thebenefit of U.S. Provisional Application Ser. No. 60/641,752, filed Jan.7, 2005, each of which is incorporated herein in its entirety for allpurposes.

TECHNICAL FIELD

This disclosure relates to image processing.

BACKGROUND

Industrial products exist in the market for three-dimensionaldigitization for various purposes. Examples include medicalapplications, entertainment industry applications (e.g.,three-dimensional gaming, filming, and animation), fashion design (e.g.,three-dimensional garment design, apparel fitting, and plastic surgery),archaeological restoration and/or preservation, forensic applications(e.g., crime scene investigation), and online commodity exhibition(e.g., online museum and online store).

There are, in general, two categories of three-dimensional digitizingtechniques: active sensing and passive sensing. Techniques belonging tothe first category, active sensing, usually emit certain energy (e.g.,light and/or sound etc.) toward the scene to be measured/observed, andreceive the reflected energy or observe the reflected pattern, makinguse of the physics law in optics or acoustics to derive the distancefrom the sensor to the object in the scene. Active sensing usually needsa complex and sophisticated optical design of the lighting components,and it usually needs controlled ambient lighting to assist in thethree-dimensional capturing. Sensors within this category are usuallylimited to sensing static scenes/objects because they usually need acertain amount of time to accomplish the scanning procedure due to thenormal requirement of physically moving certain components in thescanning systems (e.g., components for emitting lasers within thiscategory need to be moved to scan different lines of the object). Laserscanning, moiré fringe contouring, time of flight, and structuredlighting are among the active three-dimensional sensing techniques.

The techniques in the second category of passive sensing, on thecontrary, usually do not emit energy toward the scene. Instead, thesetechniques capture certain signals that are available in the scene, suchas intensity and/or color and, by analyzing these signals along withsensor configuration information, these techniques obtainthree-dimensional information for the scene. Stereovision (two or morecameras) is a typical example of passive three-dimensional sensing.

Passive sensing usually does not need a complex optical design. Forexample, a stereovision system usually takes a snapshot of thescene/object and recovers the three-dimensional information with simpledevices. Some systems also integrate more cameras in one system tocapture both three-dimensional information and color texture informationfrom the scene/object. Systems with sufficiently fast computer CPU timealso may handle dynamic scenes. To ensure the stereo cue has sufficientfeatures to match the two views, stereovision-based systems usually needto introduce some additional features onto the scene/object. Projectors(e.g., slide projector or an LCD) are often used to project suchpatterns onto the surface. In such systems, the pattern is switched onand off in order to capture both (1) the image with the superimposedfeatures and (2) the color texture image of the scene/object without thesuperimposed features. This generally requires a certain mechanism toturn the pattern on and off. In addition, in situations in which theobject of interest is a human being, illuminating patterns onto the faceof the human being may cause discomfort to the eyes.

Known stereo systems establish a correspondence between the two stereoviews. In general, there are mainly two types of methods for computingthe correspondence, or matching. The first method is a feature-basedmethod, which usually generates matches for those positions in theimages that have abundant information about the scene, such as corners,edges, and line segments. The second method is an area-based matchingtechnique, which matches the two views based on pixel similarity inlocal image regions. The feature-based method (the first method) usessurface texture feature information and generates matches for a limitednumber of pixels. The area-based method (the second method), istypically computationally more expensive, but is typically able togenerate dense matches. For three-dimensional digitizing, the higher theresolution at which the three-dimensional surface is sampled, the betterthe surface is usually captured. The feature-based stereo matchingmethod typically does not provide sufficient matched points for thispurpose. The area-based stereo matching method can typically generatesufficient numbers of three-dimensional samples on the surface, however,this method may have a long computation time, especially for highresolution capturing.

SUMMARY

At least one disclosed implementation provides a three-dimensionaldigitizer system based on a stereovision passive sensing technique. Sucha digitizer system is able to capture the scene in real time because itdoes not involve the physical movement of a component. The digitizersystem also uses an infra-red filter to project a pattern onto the scenewhich enables the system to capture both the measurement image(s) andthe texture image(s) simultaneously. These features make this systemsuitable not only for static objects, but also for three-dimensionalreconstruction of dynamic scenes. The simple and straightforwardprinciples of this system do not require complicated mechanical orelectrical design or special installation. This system works in a normaloffice environment, requiring no controlled ambient lighting, and istherefore easy to port and to use. This system may also use a referencestripe as the pattern in stereovision, and may use a described matchingpropagation technique to assist in finding a correspondence between theviews efficiently and accurately. The system also may perform filteringand smoothing using a Kalman filter and Non-Uniform Rational B-Splinesurface fitting to attempt to effectively deal with the noise fromsensors and numerical calculation.

According to a general aspect, processing images includes projecting aninfra-red pattern onto a three-dimensional object. Processing the imagesalso includes producing a first image, a second image, and a third imageof the three-dimensional object while the pattern is projected on thethree-dimensional object. The first image includes the three-dimensionalobject and the pattern, and it is a two-dimensional digital imageincluding pixels. The first image is produced by capturing at a firstcamera light filtered through an infra-red filter. The second imageincludes the three-dimensional object and the pattern, and it is atwo-dimensional digital image including pixels. The second image isproduced by capturing at a second camera light filtered through aninfra-red filter. The first and second cameras are arranged as a firststereo pair having a known physical relationship. The third imageincludes the three-dimensional object but not the pattern, and it is atwo-dimensional digital image including pixels. Processing the imagesalso includes establishing a first-pair correspondence between a portionof the pixels of the first image and a portion of the pixels in thesecond image. Processing the images further includes constructing, basedon the first-pair correspondence and the third image, a two-dimensionalimage that depicts a three-dimensional construction of thethree-dimensional object.

Implementations of the above general aspect may include one or more ofthe following features. For example, projecting the infra-red patternmay include projecting a non-random infra-red pattern. The pattern mayinclude vertical stripes. The light may be non-infrared light. The thirdimage may be produced by capturing non-filtered light at a third camera.The third camera may be a texture camera.

Establishing the first-pair correspondence may include determining acorrespondence between an initial pixel in the first image and acorresponding pixel in the second image. Establishing the first-paircorrespondence may also include determining a correspondence betweenadditional pixels in the first image and corresponding pixels in thesecond image, based on the correspondence between the initial pixel inthe first image and its corresponding pixel in the second image.

Establishing the first-pair correspondence may include determining acorrespondence between a first initial pixel located on a firstparticular horizontal line in the first image and a first correspondingpixel that corresponds to the first initial pixel. The firstcorresponding pixel may be located on the first particular horizontalline in the second image. Establishing the first-pair correspondence mayalso include determining a correspondence between additional pixelslocated on the first particular horizontal line in the first image andcorresponding pixels that correspond to the additional pixels. Thecorresponding pixels may be located on the first particular horizontalline in the second image. Establishing the first-pair correspondence mayalso include determining a correspondence between a second initial pixellocated on a second particular horizontal line in the first image and asecond corresponding pixel that corresponds to the second initial pixel.The second corresponding pixel may be located on the second particularhorizontal line in the second image. Establishing the first-paircorrespondence may also include determining a correspondence betweenadditional pixels located on the second particular horizontal line inthe first image and corresponding pixels that correspond to theadditional pixels. The corresponding pixels may be located on the secondparticular horizontal in the second image.

Establishing the first-pair correspondence may include determining acorrespondence between an initial pixel in each horizontal line in thefirst image and a corresponding pixel in each horizontal line in thesecond image. Correspondence between additional pixels in the firstimage and corresponding pixels in the second image may be determinedbased on the correspondence between the initial pixel in each horizontalline in the first image and its corresponding pixel in each horizontalline in the second image. The first initial pixel may be a centroidpattern pixel calculated from the pattern pixels in the first particularhorizontal line.

Determining the correspondence for at least one of the additional pixelslocated on the second particular horizontal line in the first image maybe based on the correspondence determined for at least one other pixellocated in the second particular horizontal line.

Determining the correspondence for at least one of the additional pixelslocated on the second particular horizontal line in the first image maybe based on the correspondence determined for at least one pixel locatedin the first particular horizontal line. The at least one pixel locatedin the first particular horizontal line may be in a common stripe edgewith the at least one of the additional pixels located in the secondparticular horizontal line.

Constructing the two-dimensional image that depicts thethree-dimensional construction may include forming a first set ofthree-dimensional points based on the first-pair correspondence andproducing a first three-dimensional surface model based on the first setof three-dimensional points.

A fourth image of the three-dimensional object may be produced while thepattern is projected on the three-dimensional object. The fourth imagemay be a two-dimensional digital image including pixels and may beproduced by capturing at a fourth camera light filtered through aninfra-red filter.

A fifth image of the three-dimensional object may be produced while thepattern is projected on the three-dimensional object. The fifth imagemay be a two-dimensional digital image including pixels and may beproduced by capturing at a fifth camera light filtered through aninfra-red filter. The fourth and fifth cameras may be arranged as asecond stereo pair having a known physical relationship. A second-paircorrespondence may be established between a portion of the pixels of thefourth image and a portion of the pixels of the fifth image.Constructing the two-dimensional image that depicts thethree-dimensional construction of the three-dimensional object mayfurther be based on the second-pair correspondence.

Constructing the two-dimensional image that depicts thethree-dimensional image may include producing a first three-dimensionalsurface model based on the first-pair correspondence, producing a secondthree-dimensional surface model based on the second-pair correspondence,and registering the first and the second three-dimensional surfacemodels. Registering the first and the second three-dimensional surfacemodels may include determining a common surface in the first and thesecond three-dimensional surface models. The common surface may be usedto produce an initial estimate for a registration matrix. The initialestimate for the registration matrix may be used to determine theclosest points between the first and the second three-dimensionalsurface models.

Producing the first three-dimensional surface model may include forminga first set of three-dimensional points based on the first-paircorrespondence and producing the first three-dimensional surface modelbased on the first set of three-dimensional points. Producing the secondthree-dimensional surface model may include forming a second set ofthree-dimensional points based on the second-pair correspondence andproducing the second three-dimensional surface model based on the secondset of three-dimensional points.

The first and second three-dimensional surface models may be integratedafter registration to produce an integrated three-dimensional surfacemodel. Texture may be provided to the integrated three-dimensionalsurface model.

According to another general aspect, a system for processing imagesincludes a first stereo camera pair including a first camera coupled toa second camera and a second stereo camera pair including a third cameracoupled to a fourth camera. The system also includes a set of fourinfra-red filters, with a separate one of the four infra-red filtersoperatively coupled to each of four cameras and a projector. The systemfurther includes a computer readable medium coupled to each of the fourcameras and to the projector. The computer readable medium includesinstructions for projecting an infra-red pattern from the projector ontoa three-dimensional object. The computer reasonable medium also includesinstructions for producing a first image, a second image, and a thirdimage of the three-dimensional object while the pattern is projected onthe three-dimensional object. The first image includes thethree-dimensional object and the pattern, and it is a two-dimensionaldigital image including pixels. The first image is produced by capturingat a first camera light filtered through an infra-red filter. The secondimage includes the three-dimensional object and the pattern and it is atwo-dimensional digital image including pixels. The second image isproduced by capturing at a second camera light filtered through aninfra-red filter. The first and second cameras are arranged as a firststereo pair having a known physical relationship. The third imageincludes the three-dimensional object but not the pattern and it is atwo-dimensional digital image including pixels. The third image isproduced by capturing light at a texture camera. The computer readablemedium also includes instruction for establishing a first-paircorrespondence between a portion of the pixels of the first image and aportion of the pixels of the second image. The computer readable mediumfurther includes constructing, based on the first-pair correspondenceand the third image, a two-dimensional image that depicts athree-dimensional construction of the three-dimensional object.

Implementations of the above general aspect may include one or more ofthe following features. For example, the projector may include alighting source capable of producing light in the visible spectrum andin the infrared spectrum. The projector may include a fifth infra-redfilter. The computer readable medium may include one or more of aprocessing device and a storage device.

According to another general aspect, a computer readable medium includesinstructions for accessing a first image, a second image, and a thirdimage captured of a three-dimensional object. The first image iscaptured while a pattern is projected on the three-dimensional object,and it includes the three-dimensional object and the pattern. The firstimage is a two-dimensional digital image including pixels. The firstimage is produced by capturing at a first camera light filtered throughan infra-red filter. The second image is captured while the pattern isprojected on the three-dimensional object, and it includes thethree-dimensional object and the pattern. The second image is atwo-dimensional digital image including pixels, and it is produced bycapturing at a second camera light filtered through an infra-red filter.The third image is captured while the pattern is projected on thethree-dimensional object, and it includes the three-dimensional objectbut not the pattern. The third image is a two-dimensional digital imageincluding pixels. The computer readable medium also includesinstructions for establishing a first-pair correspondence between aportion of the pixels of the first image and a portion of the pixels ofthe second image. The first-pair correspondence is established based onthe first and second cameras having been arranged as a first stereo pairhaving a known physical relationship while the first and second imageswere captured. The computer readable medium also includes instructionsfor constructing, based on the first-pair correspondence and the thirdimage, a two-dimensional image that depicts a three-dimensionalconstruction of the three-dimensional object.

The various aspects, implementations, and features may be implementedusing, for example, one or more of a method, an apparatus, an apparatusfor performing a method, a program or other set of instructions, anapparatus that includes a program or other set of instructions, or acomputer readable medium. The computer readable medium may include, forexample, instructions, software, images, and other data.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features will beapparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee.

FIG. 1 illustrates a three-dimensional digitizing system.

FIG. 2 illustrates a process using the three-dimensional digitizingsystem of FIG. 1.

FIG. 3A illustrates a first image and a second image including patternscaptured in a first stereo pair.

FIG. 3B illustrates a third image and a fourth image including patternscaptured in a second stereo pair.

FIG. 4 illustrates a first striped pattern imposed on one of the imagesof FIGS. 3A-3B.

FIG. 5 illustrates a second striped pattern imposed on one of the imagesof FIGS. 3A-3B.

FIG. 6 illustrates a grid that includes a set of vertical stripes andhorizontal lines.

FIGS. 7A-7B illustrate a process for locating initial matched pixels ina pattern that includes a special stripe.

FIG. 7C illustrates an intensity profile that includes a curverepresenting the intensity values of pixels located on one of thehorizontal lines of FIG. 6.

FIG. 8 illustrates a process for locating initial matched pixels in apattern that does not include a reference stripe.

FIG. 9 illustrates a centroid pixel that is not a stripe pixel.

FIG. 10 illustrates a first grid and a second grid including a centroidstripe pixel and a pixel corresponding to the centroid stripe pixel,respectively.

FIG. 11 illustrates a process for propagating disparities in two images.

FIG. 12 illustrates a process for propagating disparities in ahorizontal direction.

FIG. 13 illustrates a process for propagating disparities in a verticaldirection.

FIG. 14 illustrates a process for generating a single three-dimensionalsurface model with texture.

FIG. 15 illustrates a process for computing an initial transformationmatrix.

FIGS. 16A-16B illustrate a first three-dimensional surface model and asecond three-dimensional surface model before registration.

FIGS. 17A-17B illustrate the first and the second three-dimensionalsurface models of FIGS. 16A-16B after registration.

FIG. 18 illustrates a single three-dimensional surface model withouttexture and a single three-dimensional surface model with texture.

FIG. 19 illustrates a process for reconstructing a three-dimensionalimage of an object.

DETAILED DESCRIPTION

Referring to FIG. 1, an implementation for a three-dimensionaldigitizing system 100 is shown, which includes five cameras 102, 104,106, 108, and 110. The cameras are arranged as two stereo pairs 101 aand 101 b, with stereo pair 101 a including cameras 102 and 104, stereopair 101 b including cameras 108 and 110, and as one texture camera 106.System 100 also includes a pattern projector 112, five infrared filters114 (one on each of stereo cameras 102, 104, 108, and 110 and one onpattern projector 112), and a slide 116 having a pattern. As anillustrative example, slide 116 is shown with a vertical-line pattern.Cameras 102, 104, 106, 108, and 110 are coupled to a computer processor118 via wires 120.

The three-dimensional digitizing system 100 is able to capturethree-dimensional information of a dynamic or a static scene. System 100uses infrared lighting to illuminate an object and uses infrared filters114. System 100 illuminates the object with a desired pattern usingprojector 112, filter 114, and slide 116 containing the desired pattern.Projector 112 may include a normal light source capable of producinglight in the visible spectrum and in the infrared spectrum. The normallight is filtered so that predominately only infrared light isprojected. In another implementation, the projector may include infraredlight source without filter 114. System 100 captures the superimposedpattern using stereo cameras 102, 104, 108, and 110, which are eachequipped with one of infrared filters 114. Also, system 100 is able tosimultaneously capture the color texture of the object using texturecamera 106. Because camera 106 does not include one of the infraredfilters 114, camera 106 will receive all incoming light, includingvisible light and infrared light. However, the visible light istypically much stronger than the infrared light and, accordingly, camera106 does not visibly show the infrared pattern on the captured image.System 100 transmits this information to computer processor 118 viawires 120. Computer processor 118 may employ a process 200, describedbelow, that uses this information to create a single three-dimensionalsurface with texture. The use of infrared lighting and infrared filtersprovides an implementation that does not have a visible illumination onthe object being photographed. If the object is a face of a lone human,the lack of the visible illumination may be advantageous because thepattern will not distract the human. In addition, the use of infraredlighting and infrared filters permits all of the images to be takensimultaneously (depending on the speed, for example, of the fire-wirecard or USB interface), because the pattern need not be switched on andoff. Further, because the pattern need not be switched on and off, theimages can be taken without requiring additional control of thehardware, except for controlling the cameras to capture images.

Referring to FIG. 2, process 200 may use system 100, for example, toperform three-dimensional reconstruction of an object. Process 200includes acquiring an image (202). In one implementation that usessystem 100, as part of acquiring the image (202), a first stereo image,a second stereo image, and a texture image are produced while thepattern is projected on the three-dimensional object. The first stereoimage is produced by the stereo pair 101 a and includes a first imageand a second image of the three-dimensional object. The first image isproduced by capturing, at camera 102, light filtered through infraredfilter 114. The second image is produced by capturing, at camera 104,light filtered through infrared filter 114. The first image includes thethree-dimensional object with the pattern and may be a two-dimensionaldigital image including pixels. Similarly, the second image includes thethree-dimensional object with the pattern and is a two-dimensionaldigital image including pixels. The second stereo image is produced bythe stereo pair 101 b and includes a third image and a fourth image ofthe three-dimensional object. The third image is produced by capturing,at camera 108, light filtered through infrared filter 114 and is atwo-dimensional digital image including pixels. Similarly, the fourthimage is produced by capturing, at camera 110, light filtered throughinfrared filter 114 and is a two-dimensional digital image includingpixels.

In the above implementation, the texture image of the three-dimensionalobject is produced by capturing non-filtered light at texture camera106. The texture image includes the three dimensional object, withoutrevealing the infrared pattern, and is a two-dimensional digital imageincluding pixels.

Process 200 includes, optionally (as indicated by the dashed line aroundoperation 201), calibrating one or more, or all of the cameras 102, 104,106, 108, and 110 (201). As part of the calibration (201), one or more,typically multiple, camera parameters may be determined and modified,such as, for example, extrinsic and intrinsic parameters. Extrinsicparameters include translation and rotation of the cameras with respectto a reference coordinate system. Intrinsic parameters include camerafocal lengths, the image center, and lens distortion parameters. Thecalibration in this implementation is done during system setup, and bothintrinsic and extrinsic parameters are stored and later used in thethree-dimensional reconstruction process. System 100 may need to bere-calibrated if one of cameras 102, 104, 106, 108, and 110 is movedwith respect to the other cameras. Calibration typically involves humaninput, although it need not necessarily require human input. Otheroperations, particularly subsequent operations, in process 200 typicallywork automatically without human input.

Process 200 includes producing a first three-dimensional surface model(also referred to as a first mesh) (204). Producing the firstthree-dimensional surface model (204) may use one or more of the cameraparameters and the first stereo image. The camera parameters may be usedto rectify the first stereo image according to well-known techniques inthe art. As part of rectification in one implementation, the extrinsicand intrinsic parameters are used to compute a 3×3 rectification matrixfor each image captured in the stereo pair 101 a. The rectificationmatrices transform the two images of the stereo pair 101 a such that thecorresponding pixels in each image have the same horizontal position.The first stereo image is used to establish a first-pair correspondencebetween pixels of the first image and pixels of the second image. Thefirst-pair correspondence is used to establish a first set ofthree-dimensional points, which is in turn used to produce the firstthree-dimensional surface model (204).

Similarly, process 200 includes producing a second three-dimensionalsurface model (also referred to as a second mesh) (206). Producing thesecond three-dimensional surface model may use one or more of the cameraparameters and the second stereo image. The camera parameters may beused to rectify the second stereo image according to well-knowntechniques in the art. As described above in one implementation, theextrinsic and intrinsic parameters are used to compute a 3×3rectification matrix for each image captured in the stereo pair 101 b.The rectification matrices transform the two images captured in thestereo pair 101 b such that the corresponding pixels in each image havethe same horizontal position. The second stereo image is used toestablish a second-pair correspondence between pixels of the third imageand pixels of the fourth image. The second-pair correspondence is usedto produce a second set of three-dimensional points, which is in turnused to produce the second three-dimensional surface model (206).

As part of producing the first and the second three-dimensional surfacemodels (204 and 206), a stereo matching method may be used. Oneparticular stereo matching method includes projecting a predefinedpattern consisting of a set of stripes onto the three-dimensionalobject. The predefined pattern provides sufficient information toestablish the first-pair correspondence between stripe pixels of thefirst image and the stripe pixels of the second image. Similarly, thepredefined pattern provides sufficient information to establish thesecond-pair correspondence between stripe pixels of the third image andthe stripe pixels of the fourth image. The resolution of thereconstructed three-dimensional image of the object depends on theresolution of the stripes because the stripes are used to compute thethree-dimensional surface of the object. Thus, if more stripes are usedas part of projected pattern, more three-dimensional points on thesurface are sampled and more surface detail is captured. It is possibleto choose the proper pattern according to the complexity of the objectto be digitized. For example, a surface with less detail (e.g. afootball) needs fewer stripes than a surface with more detail (e.g., ahuman face).

In one implementation, the stereo matching includes a disparitypropagation approach. The disparity propagation approach includeslocating initial matched pixels, also referred to as seeds, in twoimages of each of the stereo pairs 101 a and 101 b. To locate theinitial matched pixels, a reference stripe may be used to provide ananchoring position. The occurrence of the reference stripe in each ofthe two images may be taken as the first matched pair of stripes. In onesuch implementation shown in FIGS. 3A-3B, a reference stripe 308 iswider than other stripes 306. In another implementation, the referencestripe 308 may have the same width as the other stripes 306.

After locating the initial matched pixels, the disparity propagationapproach propagates the initial matched pixels in one or moredirections, as described below in more detail with respect to FIGS.11-13. As part of propagating the initial matched pixels, considerationis given to the possibility that one or more depth discontinuities mayoccur. A first image 302, a second image 304, a third image 312, and afourth image 314, each includes a depth discontinuity 310 betweenchin/neck surfaces of the object (e.g., the face of FIG. 3A). The depthdiscontinuity occurs at the junction of two (or more) physical surfaceshaving different depths. The depth discontinuity presents a difficultyin propagating the initial matched pixels in the two images because thedepth discontinuity deforms the stripes, which are used in the disparitypropagation approach to match the two images. For example, the depthdiscontinuity may cause the same physical stripe to be shown as severalbroken segments in the chin/neck area, neck/clothing area, and/or nosearea of the object (e.g., the face of FIG. 3A), as further discussedbelow with respect to FIG. 4. Furthermore, the depth discontinuity maycause two stripes to join each other and form a single stripe, asfurther discussed below with respect to FIG. 5.

FIG. 4 illustrates an example of a striped pattern 400 projected ontoone of the images 302, 304, 312, and 314 without the image being shown.As illustrated, the depth discontinuity deforms the striped pattern 400,which includes stripes 402, 404, and 406. For example, stripe 402 breaksat a region 412 corresponding to a chin/neck area of the object (e.g.,the face of FIG. 3A) over which the pattern is projected. Similarly,stripe 404 breaks in a region 414 corresponding to a nose of the object(e.g., the face of FIG. 3A) over which the pattern is projected. Stripe406 includes stripe ends 416, which are adjacent to an area 420. Thearea 420 corresponds to a space between a jaw and a shoulder of theobject (e.g., the face of FIG. 3A) over which the pattern is projected.In one implementation, the disparity propagation approach takes intoaccount the depth discontinuities noted above. For example, thedisparity propagation approach does not propagate the initial matchedpixels across stripe ends 416 because the disparity propagation approachhas marked stripe ends 416 such that the stripe ends 416 stop thepropagation, as described below in more detail with respect to FIGS.7A-7C.

As noted above, the depth discontinuity may also cause two stripesegments to join each other. FIG. 5 illustrates an example of a stripedpattern 500 captured in one of the stereo cameras 102, 104, 108, and110. Again, as with FIG. 4, the striped pattern 500 has been projectedonto an image, and the image is removed to show the pattern 500 inisolation. The striped pattern 500 includes stripes 502 and 504. Stripe502 includes a stripe end 508. Stripe 502 breaks at the stripe end 508due to depth discontinuity and the lower segment of stripe 502 shifts toa lower segment 506 of stripe 504. If the initial matched pixels aresimply propagated along stripe 504 there is a high possibility ofintroducing wrong matches. This is because as part of disparitypropagation, described in more detail below with respect to FIGS. 11-13,a small window (e.g., ±3 pixels) is used to find the best disparity forthe initial matched pixels. The disparity means the difference betweencolumn positions of the initial matched pixels. The best disparityresults when the difference between the column positions of the initialmatched pixels is a minimum. However, when the depth discontinuitycauses two stripe segments join each other, the best disparity may falloutside the search window. Thus, if the initial matched pixels arepropagated along the stripe with the depth discontinuity (e.g., alongthe segment 506), incorrect matches are produced. One implementation ofthe disparity propagation approach includes several techniques foraddressing this problem, as described below in more detail with respectto FIG. 13.

As noted above, an implementation of the disparity propagation approachincludes locating the initial matched pixels. There are several methodsfor locating the initial matched pixels, depending on the type ofpattern used. In one implementation, a special reference stripe is usedto locate the initial matched pixels. In this implementation shown inFIGS. 3A-3B, the projected pattern includes the reference stripe 308,which is wider than the other stripes 306 such that the reference stripe308 is unique in the image. Thus, locating the initial matched pixels isaccomplished by locating the position of the reference stripe 308 in thetwo images.

FIG. 6 illustrates a grid 600 that includes horizontal lines 614, alsoreferred to as raster lines, and stripes 602, 604, 606, 608, and 610.Stripes 602, 604, 606, 608, and 610 are the striped pattern imposed onan object (e.g., the face of FIG. 3A). Stripe 602, also referred to as areference stripe, is different from stripes 604, 606, 608, and 610. Thereference stripe 602 is located in the center of the projected patternand has a width w′. The width w′ of the reference stripe 602 isapproximately twice the width w of stripes 604, 606, 608, and 610.Stripes 602, 604, 606, 608, and 610 are located within a distance d fromeach other. In one implementation the distance d is equal to the widthw. In another implementation, the distance d may be larger or smallerthan the width w.

Stripes 602, 604, 606, 608, and 610 may break at different locations dueto depth discontinuities. For example, the reference stripe 602 breaksat a region 611 that may correspond to a chin/neck area of the object(e.g., the face of FIG. 3A). The reference stripe 602 may also break ata neck/clothing region of the object (e.g., the face of FIG. 3A).Furthermore, the depth discontinuity may cause segments of the referencestripe 602 to join segments of another projected stripe and form asingle stripe in the image. As such, depth discontinuities may make itdifficult to recover a complete reference stripe in images 302, 304,312, and 314. However, observation shows that the “wider” property holdseverywhere along the reference stripe 602. Therefore, thischaracteristic may be utilized in locating segments of the referencestripe 602.

Shadow and occlusion normally create wide stripes in the image, whichmay appear like a reference stripe. Thus, shadow and occlusion interferewith locating the reference stripe 602. The disparity propagationapproach excludes the stripes created by shadow and occlusion from thestripes that are considered in selecting the reference stripe 602. To doso, in one implementation, the disparity propagation approach estimatesthe width w of the majority of the stripes according to awinner-takes-all scheme. The winner-takes-all scheme is well-known inthe art. The winner-takes-all scheme uses a voting strategy to determinethe width w. Based on the voting strategy, each edge stripe pixel (e.g.,stripe pixels located at the right and left edges of a stripe) casts avote about the width of its stripe. In one implementation, right edgestripe pixels look at their closest left edge stripe pixels to cast avote about the width of their local stripes. For example, the right edgestripe pixel 616 looks at the closest left edge stripe pixel 618 to casta vote about the width of stripe 604. Additional edge stripe pixels instripe 604 and the other stripes on all raster lines 614 also cast avote. The winner-takes-all scheme selects the width having the mostvotes across all stripes to be the width w for the majority of thestripes. The disparity propagation approach excludes the stripes havingwidth greater than twice the width w from the stripes that areconsidered in selecting the reference stripe 602 because these stripeare likely created by shadows and occlusion.

Features of the object (e.g., the face of FIG. 3A) that is beingphotographed may cause additional stripes to appear in images capturedin each of the stereo pairs 101 a and 101 b. For example, when theobject is the human face, eyebrows and facial hairs may cause smallstripes to appear among the projected striped pattern. The small stripesmay be referred to as noisy data, and they affect the estimation of thewidth w when an averaging method is used. The averaging method takes theaverage of the width of all the stripes on all raster lines 614 tocalculate the width w. Thus, in the averaging method the width of thenoisy data is used in calculating the width w. However, thewinner-takes-all scheme attempts to avoid the noisy data by selectingthe width w that has the most votes across all stripes. Even though thewidth of the noisy data may have some votes, it typically will not haveas many votes as the width w of the majority of the stripes. As aresult, the disparity propagation approach typically also reduces theinfluence of noisy data.

After determining the width w, the reference stripe pixels on thereference stripe 602 are located. FIGS. 7A-7B illustrate a process 700for locating the reference stripe pixels and finding the initial matchedpixels for images captured in each of the stereo pairs 101 a and 101 b.Process 700 includes extracting an intensity profile for each rasterline 614 (702). Referring to FIG. 7C, an intensity profile 701 includesa curve 703 representing the intensity values of the pixels located onone of the raster lines 614. The curve 703 includes local maximums 703a, 703 b, 703 c, and 703 d, local minimums 703 e, 703 f, 703 g, and, 703h, and a search window 705. Process 700 includes calculating localextremes (e.g., local maximums and minimums) of the intensity profilefor each raster line 614 (704). To calculate the local maximums, thesearch window 705 that is 3 pixels wide is used. The search window 705searches the curve 703 from beginning portion 703 i of the curve 703 toending portion 703 j of the curve 703, and each time the intensity ofthe pixel at the center of the search window 705 is greater than theintensity of its neighboring pixels, the location and the intensity ofthe center pixel is stored in a local maximum database. In this manner,the positions and intensities of the local maximums 703 a, 703 b, 703 c,and 703 d are determined and stored in the local maximum database.Similar to locating the local maximums, the search window 705 searchesthe curve 703 from the beginning portion 703 i of the curve 703 to theending portion 703 j of the curve 703, and each time the intensity ofthe pixel at the center of the search window 705 is smaller than theintensity of its neighboring pixels, the position and the value of thiscenter pixel is stored in a local minimum database. In this manner, thepositions and intensities of the local minimums 703 e, 703 f, 703 g, and703 h are identified.

Process 700 also includes locating k*, a local minimum in the intensityprofile on each raster line 614 that is presumed to be on the referencestripe, (706) as follows:

k _(n)*=arg_(max)(|X _(i) −X_(j)|*|((Int_(i)+Int_(j))/2)−Int_(k)|)  (Equation 1)

where i and j are indices of two neighboring local maximums of theintensity profile, X_(i) and X_(j) are positions of the two localmaximums, Int_(i) and Int_(j) are intensity values at X_(i) and X_(j),respectively, k is a local minimum between the local maximums X_(i) andX_(j), Int_(k) is the intensity at the local minimum k, and n is anindex of raster lines 614. Equation 1 essentially finds, on each rasterline 614 (that is, for each “In”), a pixel on the reference stripe 602.To locate the pixel on the reference stripe 602 for each raster line614, equation 1 calculates for each local minimum k an area bymultiplying a distance between local maximums adjacent to k times theintensity difference between the average of the local maximums and thelocal minimum. Equation 1 finds the local minimum k* that maximizes thisarea. k* represent a pixel on the reference stripe because the pixels onthe reference stripe produce the largest area since the reference stripe602 is wider and darker than other stripes. For example, FIG. 7Cillustrates that because the reference stripe 602 is 2 d wide thedistance between the local maximums 703 b and 703 c is 3 d, which islarger than the distance between other adjacent local maximums. Inaddition, because the reference stripe 602 is darker than other stripesthe intensity at k* is lower than intensity at other local minimums.

k* is taken as a candidate reference stripe pixel on each raster line614. Process 700 also includes storing k* in a database (708). Forexample, the database may be a vector R={k_(n), n=1, 2 . . . N}, whichmay include k* for each raster line 614, where N is the number of rasterlines 614 in images captured in each of the stereo pairs 101 a and 101b. The vector R, for example, includes the positions of pixels 626, 628,630, and 632, which belong to the reference stripe 602. Process 700 isperformed for all the images of each of the stereo pairs 101 a and 101b. Thus, process 700 produces four vectors R^(f,L), R^(1,R), R^(2,L),and R^(2,R). R^(1,L) and R^(1,R) include sets of candidate referencestripe pixels in the left and right images of the stereo pair 101 a,where 1 refers to the stereo pair 101 a, L refers to the left image, andR refers to the right image of the stereo pair 101 a. R² Land R². Rinclude sets of candidate reference stripe pixels in the left and rightimages of the stereo pair 101 b, where 2 refers to the stereo pair 101b, L refers to the left image, and R refers to the right image of thestereo pair 101 b.

Process 700 includes matching the reference stripe pixels in the twoimages of each of the stereo pairs 101 a and 101 b (710) to removewrongly located reference stripe pixels. Matching the reference stripepixels in the two images (710) includes matching the reference stripepixels in R^(1,L) with corresponding reference stripe pixels in R^(1,R)that have the same raster line 614 position. Every reference stripepixel in R^(1,L) should have a match in R^(1,R). For every such match amatching score is calculated for a reference stripe pixel in R^(1,L) andits corresponding pixel in R^(1,R) using a matching score function toevaluate the quality of the matched pixels, as described below in moredetail with respect to FIG. 8. The matching score is also measured forthe neighboring pixels (e.g., ±3 pixels) of the current pixelsidentified as a match. The pixels producing the highest matching score(e.g., pixels having highest similarities to the reference stripe pixelsin R^(1,L)) are taken as an actual match for the reference stripe pixelsin R^(1,L). Thus, in one implementation, seven matching scores arecalculated to determine the best match for a given reference stripepixel. The seven include one matching score for the correspondingreference stripe pixel in R^(1,R) and six matching scores for thepixels±3 pixels on each side of the corresponding reference stripepixel.

Similarly, matching the reference stripe pixels in the two images (710)includes matching the reference stripe pixels in R^(2,L) with thecorresponding reference stripe pixels in R^(2,R) that have the sameraster line position. For every such match, a matching score iscalculated to determine the quality of the matched pixels, as describedbelow in more detail with respect to FIG. 8. As noted above, thematching score is also measured for the neighboring pixels (e.g., ±3pixels) of the current pixels identified as a match. The pixelsproducing the highest matching score (e.g., pixels having highestsimilarities to the reference stripe pixels in R^(2,L)) are taken as anactual match for the reference stripe pixels in R^(2,L). Thus, in oneimplementation, seven matching scores are calculated to determine thebest match for a given reference stripe pixel. The seven include onematching score for the corresponding reference stripe pixel in R^(2,R)and six matching scores for the pixels±3 pixels on each side of thecorresponding reference stripe pixel.

The operation (710) removes wrongly located reference stripe pixels dueto noise and/or shadow caused by the depth discontinuity around the nosearea or the chin/neck area of the object (e.g., the face of FIG. 3A).For example, if a reference stripe pixel in R^(1,L) cannot find itsmatch in R^(1,R) then it is likely that the stripe pixel does not belongto the reference stripe 602, and this stripe pixel is removed fromR^(1,L). However, if the reference stripe pixel in R^(1,L) is matchedwith a corresponding reference stripe pixel in R^(1,R) then this is afurther indication that the reference stripe pixels belongs to thereference stripe.

Process 700 also includes confirming the location of the referencestripe 602 using a Ransac-based plane-fitting algorithm (712). TheRansac-based plane-fitting algorithm is well-known in the art. As partof confirming the location of the reference stripe 602 (712),three-dimensional points of the matched reference stripe pixels arecalculated. The three-dimensional points are calculated using thematched reference stripe pixels and the camera parameters, based onstereo triangulation techniques well-known in the art. This producesthree-dimensional points for multiple locations along the referencestripe 602. The reference stripe 602 is a result of a light planeintersecting with the surface of the object (e.g., the face of FIG. 3A).The light plane is formed by a lighting source and a reference verticalline in slide 116. The lighting source projects light on a left edge ofthe reference vertical line in slide 116 and forms a left edge verticalplane having an orientation that may be described as being planer in thedirection of light travel (that is, a vector normal to the plane will beperpendicular to the direction of light travel forming the plane).Similarly, the lighting source projects light on a right edge and amiddle of the reference stripe 602 and forms a right edge and a middlevertical planes. The intersection of these planes with the surface ofthe object (e.g., the face of FIG. 3A) forms corresponding points of thereference stripe 602 in the images captured in each of the stereo pairs101 a and 101 b. The three-dimensional points of the matched referencestripe pixels should belong to the middle plane because the matchedreference stripe pixels are likely located in the middle of thereference stripe 602. The Ransac-based plane-fitting algorithm (“RPA”)is used to find the middle plane parameters (e.g., the normal vector ofthe middle plane and the position of the middle plane). Operation 712uses the middle plane parameters to determine the distance of thethree-dimensional points to the plane. The three-dimensional points thatare too far from the plane are disregarded. That is, the points that aretoo far from the plane are presumed not to lie on the reference stripe.

After confirming the location of the reference stripe 602 (712), process700 includes locating the initial matched pixels in the images capturedin each of the stereo pairs 101 a and 101 b (714). In oneimplementation, locating the initial matched pixels includes (714)identifying the closest edge stripe pixels to the left of the matchedreference stripe pixels in the two images. In another implementation,locating the initial matched pixels includes (714) identifying theclosest edge stripe pixels to the right of the matched reference stripepixels in the two images. For example, referring again to FIG. 6, theclosest edge stripe pixels to reference strip pixels 626, 628, 630, and632 are edge stripe pixels 612, 634, 636, and 638. The edge stripepixels 612, 634, 636, and 638 and their corresponding pixels in theother image are identified as the initial matched pixels.

After locating the initial matched pixels (714), the initial matchedpixels are propagated in one or more directions in the images capturedin each of the stereo pairs 101 a and 101 b, as explained below withrespect to FIGS. 11-13.

In another implementation, the type of pattern used does not include aspecial stripe as a reference stripe (e.g., all the stripes have thesame width). Therefore, this implementation provides an algorithm tolocate the initial matched pixels between images captured in each of thestereo pairs 101 a and 101 b without taking advantage of the specialreference stripe. The initial matched pixels located based on thismethod do not necessarily belong to the same stripe.

FIG. 8 illustrates a process 800 for locating the initial matched pixelsaccording to an implementation in which all stripes have the same width.Process 800 includes locating a centroid pixel for each raster line 614(802) in a first-pair image and a second-pair image using an averagingfunction. The first-pair image includes the first image produced by thestereo camera 102 and the second image produced by the stereo camera104. The second-pair image includes the third image produced by thestereo camera 108 and the fourth image produced by the stereo camera110. The centroid pixel is the average of all edge stripe pixels on eachraster line 614. Process 800 includes determining whether the centroidpixel is an edge stripe pixel (806), using edge extraction and edgelinking as preprocessing operations, which are well-known in the art. Ifthe centroid pixel is not an edge stripe pixel, process 800 includessetting an edge stripe pixel closest to the centroid pixel to be thecentroid stripe pixel (806 and 808). For example, a grid 900 includesstripes 902, 904, 906, 908, and 910 and raster lines 912. Grid 900 alsoincludes a centroid pixel 914, which is located on one of the rasterlines 912. Because the centroid pixel 914 is not an edge stripe pixel,an edge stripe pixel 916 which is the closest edge stripe pixel to thecentroid pixel 914 is set to be the centroid stripe pixel.

Process 800 includes finding initial matched pixels on each raster line614 (810). As part of finding the initial matched pixels (810), thecentroid stripe pixel for each raster line 614 in the first image ismatched with a corresponding centroid stripe pixel in the second image.Similarly, the centroid stripe pixel for each raster line 614 in thethird image is matched with a corresponding centroid stripe pixel in thefourth image. For example, the same process previously described, ofstarting with the corresponding position and checking for the bestmatching score in a search window, may be used.

For example, FIG. 10 illustrates grids 1000 a and 1000 b for the purposeof finding the initial matched pixels in the first image and the secondimage. Grid 1000 a includes stripes 1002, 1004, 1006, and 1008, and 1010and raster lines 1012. The stripes 1002, 1004, 1006, 1008, and 1010 arethe stripes captured in the stereo camera 102. Grid 1000 a also includesa centroid stripe pixel p and stripe pixels n_(i) and n_(j). Thecentroid stripe pixel p is located on stripe 1006. The stripe pixelsn_(i) and n_(j) are located on stripes 1002 and 1008, respectively. Grid1000 b includes stripes 1014, 1016, 1018, 1020, and 1022, and rasterlines 1012. The stripes 1014, 1016, 1018, 1020, and 1022 are the stripescaptured in the stereo camera 104. Grid 1000 b also includes stripepixels p′, n_(i)′, and n_(j)′. The stripe pixel p′ is located on stripe1018. The stripe pixels n_(i)′ and n_(j)′ are located on stripes 1014and 1020, respectively. As a part of finding the initial matched pixels,the centroid stripe pixel p in the first grid 1000 a is matched with thecorresponding centroid stripe pixel p′ in the second grid 1000 b. Thepixels p and p′ are taken as an initial guess for the initial matchedpixels for their raster line 1012.

Process 800 includes evaluating how well the pixels p and p′ are matched(812). As part of evaluating the quality of the matched pixels p and p′(812), a normalized zero-mean cross correlation (“NZMCC”) in a localwindow (15×15) is used as a matching score function m(p, p′). NZMCC iswell-known in the art. The matching score function m(p, p′) illustratedby equation 2 is a linear combination of an intensity matching scorefunction, m_(int)(p, p′), and a gradient matching score functionm_(eg)(p, p′). The intensity matching score function m_(int)(p, p′)illustrated by equation 3 determines the similarity between pixels p andp′ in an intensity map, which includes the intensities of the pixels.The gradient matching score function m_(eg)(p, p′) illustrated byequation 4 determines the similarity between pixels p and p′ in agradient map, which includes the gradient of the pixels' intensities.The matching score function m(p, p′) is defined as follows:

m(p,p′)=αm _(eg)(p,p′)+(1−α)m _(int)(p,p′)  (Equation 2)

\m _(int)(p,p′)=Σ_(Ω)(l(p)−μ)*(l′(p′)−μ′)/σσ′  (Equation 3)

m _(eg)(p,p′)=Σ_(Ω)(E(p)−μ_(E))*(E′(p′)−μ′_(E))/σ_(E)σ′_(E)  (Equation4)

In Equations 2 to 4, Ω is a 15×15 neighborhood around pixel p in imagel, Ω′ is the corresponding neighborhood around p′ in image l′. l(p),l′(p′) are the intensity values at p and p′ in image l and l′,respectively, (μ, σ) and (μ′, σ′) are the mean and standard deviation inthe neighborhood Ω and Ω′ in l and l′, respectively, E(p) and E′(p′) arethe gradient values at p and p′ in the gradient map E and E′,respectively, and (μ_(E),σ_(E)) and (μ_(E),σ_(E)) are mean and standarddeviation in the gradient map E and E′, respectively. α is a weightfactor between 0 to 1 and depending on its value, it may emphasize theintensity or the gradient matching score function in equation 2. In oneimplementation, the value of α is 0.6 to place more weight on thegradient matching score function.

Equation 2 essentially determines the quality of the match between twopixels p and p′. There are several methods for locating an edge stripepixel p′ that produces the best match quality. In one implementation,the matching score function m(p, p′) is used to determine the edgestripe pixel p′ having the highest similarity to the centroid stripepixel p. As part of determining the edge stripe pixel p′ having thehighest similarity, the matching score function m(p, p′) is calculatedfor the centroid stripe pixel p in grid 1000 a and all the edge stripepixels in grid 1000 b located on the same raster line as p, and the edgestripe pixel that produces the highest matching score is taken as amatch for p. This implementation does not always result in the bestmatch. This implementation may result in an incorrect match because itis possible that the matching score of non-corresponding pixels to behigher than the matching score of corresponding pixels. For example, thematching score m(p, n_(j)′) may be higher than matching score m(p, p′).Thus, in this implementation, n_(j)′ instead of p′ may be selected asthe best match for p.

In another implementation, a matching-support function M is used todetermine a pixel p′ having the highest similarity to the centroidstripe pixel p. Equation 5 illustrates one implementation of amatching-support function M that is used to determine whether theneighboring edge stripe pixels (n_(i) and n_(i)′) and (n_(j) and n_(j)′)have similar matching scores as the matched pixels p and p′:

M(p,p′)=Σ_(i=1) ^(A) m(n _(i) ,n _(i)′)+Σ_(i=1) ^(B) m(n _(j) ,n_(j)′)  (Equation 5)

In Equation 5, i and j are indices of the stripes on the left and rightof p and p′, respectively, A is the number of stripes on the left of pand p′, and B is the number stripes on the right of p and p′. As part ofcalculating M(p, p′), the edge stripe pixel, n_(i), i stripes to theleft of p is located. Similarly, the edge stripe pixel, n_(i)′, lstripes to the left of p′ is located. n_(i) and n_(i)′ are likelymatched pixels because if p and p′ are a good match, then it isreasonable to assume that edge stripe pixel, n_(i), i stripes to theleft of p in grid 1000 a is matched with the edge stripe pixel, n_(l)′,i stripes to the left of p′ in grid 1000 b. The matching score for n_(i)and n_(i)′ for each stripe to the left of p and p′ is calculated andadded together.

Similarly, as part of calculating M(p, p′), the edge stripe pixel,n_(j), j stripes to the left of p is located and the edge stripe pixel,n_(j)′, j stripes to the left of p′ is located. n_(j) and n_(j)′ arelikely matched pixels because if p and p′ are a good match, then it isreasonable to assume that edge stripe pixel, n_(j), j stripes to theright of p in grid 1000 a is matched with the edge stripe pixel, n_(j)′,j stripes to the right of p′ in grid 1000 b. The matching score forn_(j) and n_(j)′ is calculated for each stripe to the right of p and p′and added together. If p and p′ are a good match, then matching scoresof the neighboring pixels is typically high, resulting in a highmatching-support for p and p′. If p and p′ are a bad match, matchingscores of the neighboring pixels is typically low, resulting in a lowmatching-support for p and p′.

Process 800 includes determining a final match for the centroid stripepixel (814). As part of determining the final match (814), a pixel p*that maximizes the matching-support function M is taken as the match forp, where

p*=arg_(max p′)(M(p,p′))  (Equation 6)

The range of p′ includes all the stripe pixels in grid 1000 b located onthe same raster line 1012 as the centroid stripe pixel p. Equation 6essentially locates a pixel having the highest similarity to p. If thematching score m(p, p*) is higher than the required threshold value,then (p, p*) is taken as an initial matched pixel for the disparitypropagation process, as described below in more detail with respect toFIGS. 11-13. In one implementation, the threshold value is 0.8.

The implementation that includes matching-support function M enhancesthe matching algorithm because it may avoid a situation in which thenon-corresponding pixels produce a higher matching score than thecorresponding pixels. As described above, it is possible that n_(j)′results in a higher matching score than the correct match p′, which mayresult in selection of n_(j)′ as a good match for p. Because p andn_(j)′ are a wrongly matched pair, the matching score for theirneighboring stripes' edge pixels will typically be low. Thus, thematching-support for p and n_(j)′ will typically be low, indicating pand n_(j)′ are not a good match. As a result, this method is designed toidentify mismatches, remove them, and produce a more robust matchingresult.

Process 800 continues to find the initial matched pixels for each rasterline 1012. After finding the initial matched pixels for all the rasterlines 1012, process 800 may employ two additional operations to furtherensure the correctness and robustness of the initial matched pixels. Thetwo operations include checking local and global supports of the initialmatched pixels s_(i) and s_(i)′. The initial matched pixels s_(i) ands_(i)′ are a pair of edge stripe pixels and include positions (x_(i),y_(i)) and (x_(i)′, y_(i)′), where x represents a column and yrepresents a row. As noted above, since the images are rectified the rowpositions y_(i) and y_(i)′ are the same. However, the column positionsx_(i) and x_(i)′ are different. The difference between x_(i) and x_(i)′is called a disparity.

Checking the local support of the initial matched pixels s_(i) ands_(i)′ includes determining the disparity of the initial matched pixelss_(i) and s_(i)′ and comparing it to a disparity of the initial matchedpixels n_(k) and n_(k)′ located on the neighboring raster lines 1012. Ifthe disparity of the initial matched pixels n_(k) and n_(k)′ is similarto the disparity of the initial matched pixels s_(i) and s_(i)′, theinitial matched pixels n_(k) and n_(k)′ located on the neighboringraster lines 1012 are counted as a supporter. If the number ofsupporters in neighborhood of the initial matched pixels s_(i) ands_(i)′ is fewer than half of the neighborhood size, the initial matchedpixels s_(i) and s_(i)′ are counted as a bad match and are discarded.Checking the local support for the initial matched pixels s_(i) ands_(i)′ helps to remove mismatches near a depth discontinuity, where thematch usually does not have strong support from neighboring matches.

Checking the global support for the initial matched pixels s_(i) ands_(i)′ includes constructing a histogram of the disparities. Thehistogram is calculated for the disparities of the initial matchedpixels, and the histogram is used to analyze a distribution of thedisparities. In the histogram, a good disparity usually exhibits strongsupport, that is, many of the initial matched pixels have suchdisparity. While, a bad disparity usually exhibits poor support, thatis, only a few of the initial matched pixels share this disparity. Thus,the histogram helps to identify and remove the initial matched pixelshaving poor support.

After finding the initial matched pixels based on one of the abovemethods, the disparity of each of the initial matched pixels ispropagated in one or more directions as explained with respect to FIGS.11-13. Referring again to FIGS. 3A-3B, the disparity propagation is usedto establish a correspondence between additional edge stripe pixels inthe first image 302 and corresponding edge stripe pixels in the secondimage 304. Similarly, the disparity propagation is used to establish acorrespondence between additional edge stripe pixels in the third image312 and corresponding edge stripe pixels in the fourth image 314.

FIG. 11 illustrates a process 1100 for disparity propagation in one ormore directions in the images 302, 304, 312, and 314. The process 1100includes identifying the depth discontinuities in the projected stripes(1102). As part of identifying the depth discontinuities (1102),explicit and implicit ending parts on each stripe are marked, and whenthe explicit and implicit ending parts are encountered during disparitypropagation, the disparity propagation routine is halted in a givendirection. Referring again to FIG. 4, the explicit ending parts includethe stripe ends 416 of stripe 406 and areas 414 and 412 of stripes 404and 402, respectively. As noted above, the stripe ends 416 indicate theexistence of the depth discontinuity in the object over which thepattern is projected (e.g., between the jaw and the shoulder of the faceof FIG. 3A). The implicit ending parts include pixels on a stripe thatare adjacent to an explicit ending part of another stripe. The implicitending part may result when two stripe segments completely join eachother due to depth discontinuity. For example, referring again to FIG.5, stripe 504 includes a pixel segment 510 which is located adjacent toan explicit ending part 508 of stripe 502. The pixel segment 510 ismarked as an implicit ending part, and it stops the disparitypropagation because pixels around the pixel segment 510 likely indicatea depth discontinuity. Accordingly, by identifying and marking theimplicit ending parts, operation 1102 attempts to effectively handle thesituation in which two stripes join each other due to the depthdiscontinuity.

Process 1100 includes three disparity propagation passes forestablishing the correspondence between additional pixels in imagescaptured in each of the stereo pairs 101 a and 101 b (1104, 1106, and1108). The first pass includes propagating the disparity in a horizontaldirection (e.g., across the stripes in both the left and rightdirections) (1104). The second pass includes propagating the disparityin a vertical direction (e.g., along the stripes in both the up and downdirections) (1106). The third pass includes propagating the disparityagain in the horizontal direction (e.g., across the stripes in both theleft and right directions) (1108).

FIG. 12 illustrates a process 1200 for propagating the disparity in thehorizontal direction in the first pass. Process 1200 includescalculating a disparity of each of the initial matched pixels on eachraster line 614 (1202). Some raster lines 614 may not have the initialmatched pixels due to depth discontinuities.

As described above, because the images are rectified the correspondingpixels in each image have the same raster line position and presumablydifferent column positions. Thus, finding correspondence for theneighboring pixels of the initial matched pixels is made along theraster lines 614 of the initial matched pixels. Process 1200 includespropagating the disparity to the left of an initial matched pixel oneach raster line 614 in a reference image (1204). The reference imagefor the stereo pair 101 a may be the first image 302 or the second image304. The reference image for the stereo pair 101 b may be the thirdimage 312 or the fourth image 314. In one implementation, the firstimage 302 is taken as the reference image for the stereo pair 101 a andthe third image 312 is taken as the reference image for the stereo pair101 b. Referring again to FIG. 6, as part of propagating the disparityto the left (1204), the edge stripe pixel 616 to the left of an initialmatched pixel 612 is located in the reference image. Propagating thedisparity to the left (1204) also establishes a correspondence betweenthe edge stripe pixel 616 in the reference image and a correspondingpixel in the other image. To establish a correspondence between the edgestripe pixel 616 in the reference image and the corresponding pixel inthe other image, the disparity of the initial matched pixel 612calculated in operation 1202 is used. The disparity of the initialmatched pixel 612 provides an initial guess for the position of thecorresponding pixel in the other image. Thus, pixel 616′ (not shown)having such position is taken as an initial match for the edge stripepixel 616. The matching score for the matched pixels (616, 616′) isdetermined. In addition, the matching score for stripe pixel 616 andeach of several neighboring pixels (e.g., ±3 pixels) of pixel 616′ isalso determined. The pixel resulting in the highest matching score(e.g., having the highest similarity to the edge stripe pixel 616) istaken as an actual match for the stripe pixel 616. The highest matchingscore is compared with a threshold and if the highest matching score isabove the threshold value, the match is considered a good match (1206).If the matching score is below the threshold value, the match isconsidered a bad match (1206). If there are three consecutive pixelsthat fail to find their matches as indicated by the low matching score,the disparity propagation routine will be halted (1208). The matchingscore may be computed using, for example, NZMCC.

If the match is successful, process 1200 includes calculating adisparity for the newly matched pixels and setting this disparity as adefault disparity (1212). Propagating the disparity to the left of theinitial matched pixels (1204) also includes locating the next edgestripe pixel 618 to the left of the previous edge stripe pixel 616 onthe reference image and establishing a correspondence between the edgestripe pixel 618 in the reference image and a corresponding pixel in theother image (1214). To establish the correspondence between the edgestripe pixel 618 and the corresponding pixel, the new default disparityis used to find a correspondence for stripe pixel 618 in a mannersimilar to operation 1204. The disparity propagation continues from thenewly matched pixel and stops when an edge stripe pixel belongs to theexplicit or implicit ends of the stripe (1216 and 1218).

After the disparity propagation to the left of the initial matched pixelis completed, the disparity also propagates to the right side of theinitial matched pixel in the same manner. Process 1200 is repeated foreach raster line 614. As a result, process 1200 provides a set ofmatched horizontal bands in images captured in each of the stereo pairs101 a and 101 b, in which the matched bands presumably follow the edgesof the stripes.

FIG. 13 illustrates a process 1300 for propagating the disparity in thevertical direction in the second pass. From the previous horizontaldisparity propagation most of the stripes may have portions of theiredges matched. For example, referring again to FIG. 6, the referencestripe 602 includes an unmatched segment N and matched edge stripepixels 634 and 638. Process 1300 includes locating unmatched segment Nin the reference image for every stripe edge in the striped pattern(1302). The unmatched segment N includes ends N1 and N2. Process 1300includes locating the matched edge stripe pixels 634 located above theend N1 and the matched edge stripe pixel 638 located below the end N2(1304). Process 1300 includes comparing the disparities of the matchededge stripe pixels 634 and 638 (1306). If the difference betweendisparities of the matched edge stripe pixels 634 and 638 is above athreshold, the disparities are not propagated along the reference stripe602 (1307 and 1308), and this segment N is marked as uncertain segment.In one implementation the threshold is 3. If the disparities of thematched pixels is below the threshold (e.g., less than 3), process 1300includes finding a correspondence between unmatched segment N in thereference image and a corresponding segment in the other image based onthe disparities of the matched edge stripe pixels 634 and 638 (1307 and1310). In one implementation, to find the correspondence betweenunmatched segments N in two images the propagation is done from thecenter of the segment N toward two ends N1 and N2. Alternatively, thepropagation may be done from top to bottom of the segment N or viceversa. The process 1300 is repeated for the other stripes 604, 606, and608. In one implementation, process 1300 may begin from the top left ofthe striped pattern and move to the bottom right of the striped pattern.Alternatively, process 1300 may begin from the top right of the stripedpattern and move to the bottom left of the striped pattern. As a result,the unmatched segments N in each of the stripes may be matched with acorresponding segment in the other image.

The third pass can be performed after the previous two iterations of thedisparity propagation. The third pass attempts to match unmatched stripesegments, such as those uncertain segments mentioned above with respectto the second pass. The third pass uses the same process (process 1200)as in the first pass. An as-yet unmatched stripe may have a horizontalneighbor that was matched in the second pass, and that newly matchedneighbor may provide a disparity value allowing the as-yet unmatchedpixel to be matched.

After the three passes, a final match map may be obtained indicatingwhich edge stripe pixels are matched. Through propagation processes 1200and 1300 the search space for finding a correspondence for each stripepixel has been small (e.g., ±3 pixels). Thus, the disparity propagationapproach reduces computation demand. Without the disparity propagationapproach, matching a pixel in the first image with a corresponding pixelin the second image may require a lot larger searching window alongraster line 614 in the second image to find the corresponding pixel.This is because most of the pixels along raster line 614 in the secondimage may need to be evaluated to find a potential match for the pixelin the first image. Thus, the search window for each pixel is typicallymuch larger than the search window in the disparity propagationapproach, resulting in a higher computation demand.

Referring again to FIG. 11, the matches that have been established sofar are in integer precision. The matched pixels corresponding to anapproximately flat surface of the object may have the same integerdisparity. In the three-dimensional image these matched pixels appear atthe same depth even though they have slightly different depths in theobject. Thus, the reconstructed three-dimensional image may not besmooth and continuous because with integer disparity the transition indepths of the object is typically only shown between matched pixelshaving considerable depth difference. This may produce a stair-steppedreconstruction. In order to construct a smooth and continuous surface(1110), process 1100 includes, in one implementation, further refiningthe matches into sub-pixel precision. The sub-pixel precision improvesthe depth disparity resolution; thus, the matched pixels correspondingto approximately flat surfaces appear at their actual correspondingdepths in the reconstructed three-dimensional image. As part of refiningthe matches into sub-pixel precision, a curve-parabola-fitting method isused. According to the curve-parabola-fitting method, if a pixel (x, y)is matched to (x′, y) with matching score m₀, then a matching score m₁between (x, y) and (x′−1, y) and a matching score m₂ between (x, y) and(x′+1, y) are also computed using equation 2, as described above withrespect to FIG. 8. As part of constructing the smooth and continuoussurface (1110), a parabola is fitted into the three points (x′−1, m₁),(x′, m₀) and (x′+1, m₂), and a maximum m on the parabola and acorresponding x* are located. The matched pixel (x*, y) is taken as thefinal match for (x, y). With the sub-pixel precision of the matches, theoperation 1110 produces a smooth three-dimensional surface.

In the final match map, there are occasionally some mismatched pixels,which present themselves as spikes on the surface. Process 1100 includesperforming a mismatch detection to remove the mismatched pixels (1112).The detection may be based on a three-dimensional mesh, which will bedescribed with respect to FIG. 14. For each matched pixel, neighboringpixels are located and corresponding three-dimensional points for thematched pixel and its neighboring pixels are computed using stereotriangulation techniques. If the distance between the correspondingthree-dimensional point of the matched pixel and the correspondingthree-dimensional points of its neighboring pixels is larger than apredefined threshold, process 1100 considers the matched pixel as amismatch. For example, let q_(i,j) (=1, 2, . . . N_(i)) be theneighboring pixels of p_(i), the pixel in consideration, Q_(i,j) andP_(i) are the corresponding three-dimensional points. Then, process 1100determines whether the distance between P_(l) and Q_(i,j), for each j,is larger than a predefined threshold. If a ratio of the number of largedistance neighbors of P over the total number of neighbors P_(i) islarge, process 1100 may consider this as a point resulting from amismatch.

FIG. 14 illustrates a process 1400 for generating a singlethree-dimensional surface model with texture. Process 1400 includesgenerating a three-dimensional point cloud for each of the stereo pairs101 a and 101 b (1402). Generating the three-dimensional point cloud mayuse stereo triangulation. The stereo triangulation, which is well-knownin the art, uses the disparity of the matched pixels and the cameraparameters to compute the three-dimensional point cloud.

Process 1400 includes generating and smoothing a three-dimensionalsurface model for each of the stereo pairs 101 a and 101 b (1404). Thethree-dimensional surface model is generated for each of the stereopairs 101 a and 101 b using the three-dimensional point cloud, accordingto known techniques such as three-dimensional meshing.

In one implementation, as part of generating the three-dimensionalsurface model, a Delaunay triangulation algorithm, which is well-knownin the art, is used to build triangulation for the matched pixels in thereference images. The corresponding three-dimensional points then form athree-dimensional mesh.

The three-dimensional point cloud is, in general, noisy due to thestereo matching process as well as numerical error in the stereotriangulation. Moreover, the three-dimensional mesh is, in general,irregular because the matched pixels are extracted from the stripes,which deform when projected onto the surface due to the depthdiscontinuities. Hence, triangles in the mesh might be very different inshape and size. Thus, one implementation includes building a newthree-dimensional surface model (e.g., mesh), which conforms to imagegrids at a given resolution.

In building the new three-dimensional surface model, a rectangular gridis imposed on the image plane, which is a standard procedure in the art.The new three-dimensional surface model is further processed usingKalman filtering and smoothing. This procedure is well-known in the art.As a result of re-meshing and Kalman filtering and smoothing, athree-dimensional surface model is produced whose correspondingtwo-dimensional surface model conforms to the image grids and noise issuppressed and the surface is smoothed. When it is necessary, furthersmoothing may be obtained by fitting a NURB surface into thethree-dimensional point cloud obtained in the above process, which iswell-known in the art.

Referring again to FIG. 14, operation (1404) produces twothree-dimensional surface models (one for each of the stereo pairs 101 aand 101 b). Process 1400 uses the two three-dimensional surface modelsto generate a single integrated three-dimensional surface model forstereo pairs 101 a and 101 b (1406).

Referring again to FIG. 2, generating the single integratedthree-dimensional surface model (1406) includes registering the firstand the second three-dimensional surface models (208). Registering thefirst and the second three-dimensional surface models is well-known inthe art. Registration refers to a procedure of aligning data from thestereo pairs 101 a and 101 b, which might have their own coordinatesystems, so that the data can be transformed into a single coordinatesystem. Even when the stereo pairs 101 a and 101 b are calibrated usingthe same reference system, the data obtained by the stereo pairs 101 aand 101 b might have discrepancies due to errors in the calibrationprocedure or in the stereo pairs 101 a and 101 b themselves.

Registering the first and the second three-dimensional surface models(208) aligns the first three-dimensional surface model with the secondthree-dimensional surface model. In one implementation, to align thefirst and the second three-dimensional surface models, a rigidtransformation matrix (R,t) between the first and the secondthree-dimensional surface models is computed so that, by applying thismatrix either to the first or the second three-dimensional surfacemodel, one three-dimensional surface model is aligned with the otherthree-dimensional surface model.

The computation of the rigid transformation matrix (R,t) may involve twooperations. First, an initial transformation matrix is computed, whichroughly aligns the two three-dimensional surface models. Second,Iterative Closest Point (“ICP”) is used to refine the initialtransformation matrix iteratively and finally align the first and thesecond three-dimensional surface models.

Referring to FIG. 15, a process 1500 may be used for computing theinitial transformation matrix. Process 1500 includes identifying aregion of interest in the first and the second three-dimensional surfacemodels (1502). Referring again to FIGS. 3A-3B, the images captured ineach of the stereo pairs 101 a and 101 b do not exactly cover the samephysical surface regions because each camera is at a different location.Each image has an overlapping area with the other images, and each imagealso has an area that is only covered by that image. In FIGS. 3Aa-3B,the overlapping area includes a region covering the nose of the person'sface. Accordingly, the region covering the nose may be selected as theregion of interest in the first and the second three-dimensional surfacemodels. Because triangulation technique is used to create the first andthe second three-dimensional surface models, each point in the first andthe second three-dimensional surface models is covered by a triangle.The triangle corresponding to the nose, also referred to as a seedtriangle, is identified as the triangle falling in the central regionand is closest to the cameras.

Process 1700 includes expanding the region of interest (1504). As partof expanding the region of interest, triangles sharing same edges withthe seed triangle are located. The expansion continues by taking moretriangles sharing edges with those already in the region of interestuntil the total area of this cluster of triangles reaches a predefinedamount. In one implementation the predefined area is 10,000 mm². As aresult, expanding the region of interest produces two regions (1504),which give rough correspondences between the two three-dimensionalsurface models.

Process 1500 also includes finding an initial transformation matrixbetween the first and the second three-dimensional surface models(1506). Finding the initial transformation matrix is based on the tworegions of interest found above. Let the vertices in the two matchedregions be RG₁={p_(1i), i=1, 2 . . . n₁} and RG₂={p_(2j), j=1, 2 . . .n2}, respectively, where n₁ and n₂ are the number of vertices in the tworegions, respectively. RG₁ and RG₂ are used to compute an initialtranslation vector T₀ and an initial rotation matrix R₀. The centroidsC₁ and C₂ of RG₁ and RG₂ are computed, and T₀=C₂−C₁. Next let M_(1,cov)and M_(2,cov) be the covariance matrices of the RG₁ and RG₂. Then theeigenvectors of these two matrices are used to form the initial rotationmatrices R₁ and R₂, which are the rotation matrices of the two pointsets with respect to the global coordinate system. Then, the initialrotation matrix between the RG₁ and RG₂ is estimated as R₀=R₂*R⁻¹ ₁. R₀is the initial rotation matrix between the two point clouds. The initialtransformation matrix M₀ then comprises of the initial translationvector T₀ and the initial rotation vector R₀ (M₀={R₀, T₀}).

Then, the ICP algorithm is used, which takes M₀ as an initial value andfurther refines M₀ matrix. The output of the ICP finally aligns the twothree-dimensional surface models.

Referring to FIGS. 16A-16B, a first three-dimensional surface model 1601and a second three-dimensional surface model 1602 are illustrated beforeregistration. Referring to FIG. 16A, the first three-dimensional surfacemodel 1601 and the second three-dimensional surface model 1602 appear inblack and white and each include black and white stripes. The firstthree-dimensional surface model 1601 is shown with lighter gray-scalelines, such as those shown under the “chin” and pointed to by thereference numeral 1601. The second three-dimensional surface model 1602is shown with darker gray-scale lines, such as those shown on the “jaw”and pointed to by reference numeral 1602. Before the registration, thestripes of the first three-dimensional surface model 1601 and the secondthree-dimensional surface model 1602 appear in different positions. Forexample, the reference stripes 1603 of the first and the secondthree-dimensional surface models appear adjacent to each other ratherthan in the same position. Thus, the gaps between the stripes are verysmall, and the striped pattern is not distinctive. Referring to FIG.16B, the first three-dimensional surface model 1601 appears in a bluecolor and the second three-dimensional surface model 1602 appears in ayellow color. Before the registration, the yellow stripes appear inbetween the blue stripes.

FIGS. 17A-17B illustrate a first three-dimensional surface model 1601and a second three-dimensional surface model 1602 after registration.Referring to FIG. 17A, the first three-dimensional surface model 1601and the second three-dimensional surface model 1602 appear in black andwhite. As described above with respect to FIG. 16A, the firstthree-dimensional surface model 1601 is shown with lighter gray-scalelines, such as those shown under the “chin” and pointed by the referencenumeral 1601. The second three-dimensional surface model 1602 is shownwith darker gray-scale lines, such as those shown on the “jaw” andpointed to by reference numeral 1602. As shown, registration aligns thevertical stripes of the first three-dimensional surface model 1601 withthe vertical stripes of the second three-dimensional surface model 1602.For example, the reference stripes 1603 in both three-dimensionalsurface models 1601 and 1602 overlap and form a more distinctive stripethan in FIGS. 16A-B. Thus, the gaps between the stripes are larger inFIGS. 17A-B than the gaps between the stripes shown in FIGS. 16A-B.Referring to FIG. 17B, the first three-dimensional surface model 1601and the second three-dimensional surface model 1602 appear in color. Thefirst three-dimensional surface model 1601 appears in a blue color andthe second three-dimensional surface model 1602 appears in a yellowcolor. After the registration, the blue and yellow stripes overlap eachother creating large openings between the stripes and making the stripedpattern more distinctive. For example, the chin area 1605 is bettershown in FIGS. 17A-17B than in FIGS. 16A-16B.

Referring again to FIG. 2, integrating the first and the secondthree-dimensional surface models (210) combines the registered first andthe second three-dimensional surface models and generates a singlesurface model 2202, as shown in FIG. 18. Integrating the first and thesecond three-dimensional surface models is well-known in the art.

Referring again to FIG. 2, process 200 includes providing texture to theintegrated three-dimensional surface model (212). Providing texture tothe integrated three-dimensional surface model is well-known in the art.In one implementation, as part of providing texture to the integratedthree-dimensional surface model, the camera parameters, the image fromthe texture camera 106, and the integrated three-dimensional surfacemodel are used. Because the integrated three-dimensional surface modelis in the same coordinate system as the texture camera 106, each of thethree-dimensional points in the surface model is simply projected ontothe image plane to produce texture coordinates. Thus, providing textureto the integrated three-dimensional surface model (212) produces asingle three-dimensional surface with texture (1508). The singlethree-dimensional surface with texture is provided to users by, forexample, rendering (214) or exporting (216) the three-dimensionalsurface with texture through operations 214 and 216. Referring to FIG.20, three views of the final textured surface model 2204 areillustrated.

Referring to FIG. 19, in another implementation, a process 1900 includesprojecting an infrared pattern onto a three-dimensional object (1902)and producing a first image and a second image while the pattern isprojected on the three-dimensional object (1904). The first image may beproduced by capturing, at camera 102, light filtered through infraredfilter 114. The second image may be produced by capturing, at camera104, light filtered through infrared filter 114. The first imageincludes the three-dimensional object with the pattern and may be atwo-dimensional digital image including pixels. Similarly, the secondimage includes the three-dimensional object with the pattern and may bea two-dimensional digital image including pixels. Process 1900 includesestablishing a first-pair correspondence between a portion of the pixelsof the first image and a portion of the pixels of the second image(1906).

Process 1900 includes producing a third image of the three-dimensionalobject while the pattern is projected on the three-dimensional object(1908). In one implementation, the third image may be a texture imageand is produced by capturing non-filtered light at texture camera 106.The texture image includes the three dimensional object, withoutrevealing the infrared pattern, and may be a two-dimensional digitalimage including pixels. Process 1900 includes constructing a twodimensional image based on the first-pair correspondence and the thirdimage (1910).

Implementations, or features of implementations, may be implemented, atleast in part, in a variety of devices. For example, a compact disk(CD), a processing device, or other computer readable medium may containa program, instructions, or code segments for implementing any of themethods disclosed. Furthermore, a tool may be provided for implementingany of the methods disclosed. The tool may include, for example, acomputer-readable medium, a processing device, a camera, a projector, ora combination of these and possibly other components. A processingdevice may include, for example, a processor, a computer, a programmablelogic device, or an integrated circuit.

Implementations also may be implemented using portable devices. Examplesinclude a portable computer or other processing device; a portabletelephone; a personal digital assistant; a messaging device such as, forexample, a pager or a portable e-mail device (such as, for example, aBlackberry®); a portable music player such as, for example, an iPod®; oranother electronic portable messaging, entertainment, organization, orgaming device.

Although the implementations disclosed project a vertical stripedpattern onto the three-dimensional object (e.g., the face of FIG. 3A),other patterns may be projected onto the three-dimensional object. Forexample, a horizontal line pattern, a diagonal line pattern, and/orconcentric circles may be projected onto the three-dimensional object.

Finally, various technologies may be used, combined, and modified toproduce an implementation, such technologies including, for example, avariety of hardware, software, firmware, integrated components, discretecomponents, processing devices, memory or storage devices, communicationdevices, lenses, filters, display devices, and projection devices.

In addition, while the systems and methods disclosed generally have beendescribed in the context of constructing a 3D image of a face, theability to construct 3D images of other objects is contemplated.

A number of implementations have been described. Nevertheless, it willbe understood that various modifications may be made. For example,elements of different systems, processes, and other implementations maybe combined, supplemented, modified, rearranged in order, or removed toproduce additional implementations. Further, various technologies may beused, combined, and modified to produce an implementation, suchtechnologies including, for example, a variety of hardware, software,firmware, integrated components, discrete components, processingdevices, memory or storage devices, communication devices, lenses,filters, display devices, and projection devices. Accordingly, otherimplementations are within the scope of the following claims.

1. A computer-readable medium encoded with a computer program comprisinginstructions that, when executed, operate to cause a computer to performoperations comprising: marking explicit and implicit ending parts on astripe projected onto an object; performing a disparity propagationroutine in a horizontal direction, a vertical direction, and thehorizontal direction, respectively, further comprising halting thedisparity propagation routine in the horizontal or vertical directionbased on encountering the explicit or implicit ending parts;constructing a continuous three dimensional surface based on theperformed disparity propagation; and performing a mismatch detectionusing the constructed three dimensional surface if a distance betweenmatched pixels is greater than a predefined threshold.
 2. Thecomputer-readable medium of claim 1, wherein the explicit ending partscorrespond to an end of the stripe or a break in the stripe, and whereinthe implicit ending parts include a pixel on the stripe adjacent to oneof the explicit ending parts of a second stripe, resulting when thestripe and the second stripe join each other due to a depthdiscontinuity.
 3. The computer-readable medium of claim 1, wherein thestripe is projected onto the object using infrared light.
 4. Thecomputer-readable medium of claim 1, further comprising instructionsthat, when executed, operate to cause the computer to perform operationscomprising: establishing a correspondence between an edge stripe pixelin a first image and a corresponding edge stripe pixel in a secondimage, wherein the disparity propagation routine is performed based onestablishing the correspondence.
 5. The computer-readable medium ofclaim 1, wherein: the object is a user's body; and an explicit endingpart is marked between a jaw and a shoulder of the user's body.
 6. Thecomputer-readable medium of claim 1, wherein performing the disparityroutine in the horizontal direction further comprises propagating adisparity along a raster line.
 7. The computer-readable medium of claim1, wherein performing the disparity routine in the horizontal directionfurther comprises propagating a disparity across the stripe in both leftand right directions.
 8. The computer-readable medium of claim 1,wherein the disparity routine is performed in the horizontal direction,the vertical direction, and the horizontal direction, respectively.
 9. Acomputer-implemented method comprising: marking explicit and implicitending parts on a stripe projected onto an object; performing adisparity propagation routine in a horizontal direction, a verticaldirection, and the horizontal direction, respectively, furthercomprising halting the disparity propagation routine in the horizontalor vertical direction based on encountering the explicit or implicitending parts; constructing a continuous three dimensional surface basedon the performed disparity propagation; and performing a mismatchdetection using the constructed three dimensional surface if a distancebetween matched pixels is greater than a predefined threshold.
 10. Adevice comprising a processor configured to: mark explicit and implicitending parts on a stripe projected onto an object; perform a disparitypropagation routine in a horizontal direction, a vertical direction, andthe horizontal direction, respectively, further comprising halting thedisparity propagation routine in the horizontal or vertical directionbased on encountering the explicit or implicit ending parts; construct acontinuous three dimensional surface based on the performed disparitypropagation; and perform a mismatch detection using the constructedthree dimensional surface if a distance between matched pixels isgreater than a predefined threshold.
 11. A computer-readable mediumencoded with a computer program comprising instructions that, whenexecuted, operate to cause a computer to perform operations comprising:detecting first and second stripes projected onto an object, in both offirst and second images; extracting, for a raster profile and from thefirst or second images, an intensity profile including a curverepresenting intensity values of first and second pixels located on theraster line associated with the first and second stripes, respectively;determining a local minimum in the intensity profile amongst the firstand second pixels; designating the first or second stripe as a referencestripe based on the determined local minimum; matching the referencestripe in the first and second images; and constructing a threedimensional surface based on matching the reference stripe in the firstand second images.
 12. The computer-readable medium of claim 11, whereinthe first and second stripes are projected onto the object usinginfrared light.
 13. The computer-readable medium of claim 11, furthercomprising instructions that, when execute, operate to cause thecomputer to perform operations comprising: controlling a disparitypropagation along the reference stripe, wherein the three dimensionalsurface is constructed further based on controlling the disparitypropagation.
 14. A computer-implemented method comprising: detectingfirst and second stripes projected onto an object, in both of first andsecond images; extracting, for a raster profile and from the first orsecond images, an intensity profile including a curve representingintensity values of first and second pixels located on the raster lineassociated with the first and second stripes, respectively; determininga local minimum in the intensity profile amongst the first and secondpixels; designating the first or second stripe as a reference stripebased on the determined local minimum; matching the reference stripe inthe first and second images; and construct a three dimensional surfacebased on matching the reference stripe in the first and second images.15. The method of claim 14, wherein the first and second stripes areprojected onto the object using infrared light.
 16. The method of claim14, further comprising: controlling a disparity propagation along thereference stripe, wherein the three dimensional surface is constructedfurther based on controlling the disparity propagation.
 17. A devicecomprising a processor configured to: detect first and second stripesprojected onto an object, in both of first and second images; extract,for a raster profile and from the first or second images, an intensityprofile including a curve representing intensity values of first andsecond pixels located on the raster line associated with the first andsecond stripes, respectively; determine a local minimum in the intensityprofile amongst the first and second pixels; designate the first orsecond stripe as a reference stripe based on the determined localminimum; match the reference stripe in the first and second images; andconstruct a three dimensional surface based on matching the referencestripe in the first and second images.
 18. The device of claim 17,wherein the first and second stripes are projected onto the object usinginfrared light.
 19. The device of claim 17, wherein the processor isfurther configured to: control a disparity propagation along thereference stripe, wherein the three dimensional surface is constructedfurther based on controlling the disparity propagation.
 20. The deviceof claim 17, further comprising a camera configured to capture the firstand second images.